Repository navigation
feat(budget-breaker): wake on agent workflow runs, keep the schedule as a backstop - #2293
Conversation
…as a backstop The breaker was schedule-only so that only default-branch code runs with its write credential, but GitHub delivers few scheduled ticks here (two in the first 13 hours after it merged), so it could neither trip nor resume in time. - Triggers are now exactly schedule and workflow_run (completed) for the workflows in this repository that can post a dev-lead quota marker, plus the retry sweep as one more wake-up. workflow_run runs the default branch's workflow file and commit; the triggering run is never read: no reference to the event context anywhere in the workflow, no ref or repository on the checkout, and the script reads the event's name only. - No job-level filter on the triggering run's conclusion: it would need the event context, and a run that only skips may take the concurrency group's one pending place from a run that carries news. - Early exit before the organisation scan for a workflow_run wake-up on a fleet that is not paused, when an earlier run was created in the same clock slot (BUDGET_BREAKER_MIN_INTERVAL_MIN, default 2) and this repository shows no quota marker created in that slot. Never on a scheduled run, never while paused, and never when an input could not be read. The earlier runs are listed with the read credential; permissions are unchanged. - Every run logs and summarises what woke it and how long ago the workflow last ran to a conclusion. - AGENTS.md: the trigger rule, why workflow_run keeps the default-branch-only property, measured reaction time and resume delay, the cross-repository limit, and one more arming precondition. The interaction-model row is now Class 2 with a backstop timer. Refs #2259 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 17 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request introduces an 'early exit' mechanism for the budget breaker workflow to reduce unnecessary organization scans when multiple runs occur within the same time slot. It adds a 'workflow_run' trigger to the workflow, implements logic to check for recent runs and quota markers, and updates the documentation and test suite to reflect these changes. I have kept the reviewer's comments as they provide actionable feedback regarding test robustness and file extension handling.
CodeAnt Nitpicks2 code suggestions1. When another organization repository gets a quota marker after the prior scan, this local-only check can skip it; the marker may expire before another full scan.Logic error · 2.
|
|
dev-lead is withholding action on this item. It is labeled To re-enable automated pickup: remove the |
QA Lead — test-risk advisoryRisk tier: LOW. The breaker is still disarmed, so in production this change only alters when it wakes, what it logs and how many runs it costs. The new trust boundary (an untrusted What I'd shore up (highest leverage first):
Escalate? No. Advisory only — I comment, I do not change code. Ground: BMAD Test Architecture (bmad-tea v1.19.0). Opt out on this item with the qa-lead:hands-off label. |
CodeAnt PR Risk: Medium Risk
Assessed commit: |
… exit The early exit counted any earlier run that reached a conclusion. A run that failed may have ended before its organisation scan, so every later wake-up in that slot exited early and the slot got no decision. Only a run that succeeded covers a slot now. From the QA advisory and CodeAnt on the pull request. Tests: the failed-run case end to end, and the slot's edges (created exactly on the slot start, one second before it, the last second, and a boundary between the runner's clock and the API's). The workflow-name test also reads .yaml. Refs #2259 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review round 1, head
|
|
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-10-11T18:19:10Z. |



Problem
The budget breaker is schedule-only so that only default-branch code ever runs with its write credential. But GitHub delivers almost none of this repository's scheduled ticks: in the first 13 hours after the breaker merged it ran twice (08:35 and 15:03 UTC on 2026-10-11), 6.5 hours apart, against a 10-minute cron. A control that may not run for hours can neither pause the fleet during a quota outage nor, once armed, lift its own pause on time. This blocks arming ("Auto-pause companion", #1565).
Design
The maintainer agreed in principle on 2026-10-11: keep the schedule as a backstop and add a
workflow_runtrigger.Trigger.
workflow_run,types: [completed], on three workflows in this repository:pr-review's workflows are deliberately absent: its rate-limited marker is not quota evidence.
Why this keeps the default-branch-only property. GitHub's "Events that trigger workflows" page (read 2026-10-11) says of
workflow_run: "This event will only trigger a workflow run if the workflow file exists on the default branch", withGITHUB_SHAthe "Last commit on default branch". The same page warns that such a workflow "is able to access secrets and write tokens, even if the previous workflow was not". So the triggering run is treated as untrusted and is never read:if:,env:,with:,run:orconcurrency:.ref:and norepository:. No artifact is downloaded. The script reads the event's name from the runner for its log and nothing else of the event.What an outsider gains. Anyone who can make a listed workflow's run complete here (a comment, a fork pull request, a pull request adding a workflow that carries a listed name) now chooses when the breaker runs. They choose nothing else: quota markers still count only from owners, members and collaborators, and the decision is idempotent.
Cost control: an early exit in the script, not a job-level
if:. About half of today's Dev-Lead Agent completions are runs cancelled in their lane. Two reasons there is no conclusion filter:skipped. Filteringskippedwould drop exactly the runs a resume relies on.Instead, a run woken by
workflow_runon a fleet that is not paused exits before the organisation scan when both hold: an earlier run of this workflow was created in the current clock slot (BUDGET_BREAKER_MIN_INTERVAL_MIN, default 2 minutes,0switches it off), and this repository shows no quota marker created in that slot. It never applies to a scheduled run, never while the fleet is paused, and never when an input could not be read. The rule is clock slots rather than "the previous run finished less than N minutes ago" because a run that exits early is itself a previous run; under a steady stream of wake-ups that rule would exit early for ever.Visibility. Every run logs, and puts in its summary, which event woke it and how long ago the workflow last ran to a conclusion.
Risk
check_run. If the breaker's own job produced acheck_runevent that started Dev-Lead Agent, its completion would wake the breaker again. In the two real breaker runs so far (job completed 08:35:45 and 15:05:43 UTC) nocheck_run-triggered Dev-Lead Agent run followed within a minute. Two data points; see Monitoring.workflow_runfires when the triggering run endedskipped. It was observed here forcancelledruns (8 of 8 cancelled Auto-rebase runs were followed by theirworkflow_runconsumer within 3 seconds). Theskippedcase is what wakes a resume; it is added to the arming preconditions inAGENTS.mdand can be observed after merge without a pause.AGENTS.mdstates this with the measurements.BUDGET_BREAKER_ARMEDis unset, and this change does not claim arming is unblocked: the live checks on "Budget breaker: close the gaps found in review before it is armed" (Budget breaker: close the gaps found in review before it is armed (race with a human pause, pagination, hold on a bad timestamp, lost notice, live API checks) #2251) are still outstanding.Test plan
tests/dev-lead/unit/test_budget_breaker_writer.bats: 97 tests (83 before; 3 new static, 11 new behaviour).scheduleandworkflow_run; every other trigger is rejected, as aretypes: [requested]and abranches:filter.name:of a workflow file here, and the list is pinned.ref:orrepository:on the checkout; no artifact download; the scripts read only the event's name.test_budget_breaker.bats39 of 39 andtests/test_fleet_pause_coverage.bats60 of 60, unchanged.workflow_dispatch; checkout the triggering SHA; a jobif:on the triggering conclusion; early exit moved ahead of the paused check): each turned the expected test red, and the suite returned to 97 of 97.actionlintandshellcheck --severity=warning -xclean.Rollback
Revert the squash commit: the workflow goes back to schedule-only. Faster, without a merge: set the repository variable
BUDGET_BREAKER_MIN_INTERVAL_MINto a large value to make nearly every event-woken run exit early (this does not stop the runs). The breaker is disarmed either way, so a rollback changes logging only.Monitoring
workflow_run. Their summaries show "Triggered by" and "Previous run".workflow_runbreaker run that follows a Dev-Lead Agent run which endedskipped(arming precondition 6).Closes #2259
🤖 Generated with Claude Code